A color image captured by a single image sensor is typically a “mosaiced image”. This is commonly produced by using a color filter array (CFA), on which the pixels are arranged in the Bayer pattern [1], such that only one of the three primary color components (i.e., R, G, or B) is recorded at each pixel location. Demosaicing is the reverse process of mosaicing, which restores an approximate full color image based on the acquired mosaiced image. One strategy commonly practiced in the majority of demosaicing methods [2-7] is to interpolate the missing pixels on the G channel first, for this channel has the highest sampling density among all three channels in the Bayer pattern; to be exact, the number of pixels with known values is exactly twice that of the R channel and of the B channel, respectively. The missing pixels on the R and B channels are then restored by interpolating the color component difference fields R-G and B-G, respectively. This is based on the observation that the color-component differences are usually locally smooth.
Recently, a new demosaicing approach was proposed in [8], which is based on so-called residual interpolation (RI). Instead of conducting interpolation on the color-component difference (CD) fields, the demosaicing algorithm as described in [8] performs interpolation over the residual fields. Note that a residual incurred at each pixel is the difference between a known color value (i.e., ground truth) and its estimated value. If the estimated values are sufficiently close to the ground truth, the resultant residual field will be much smoother than the color-component difference field, especially when there is a sharp color transition. This indicates that interpolation conducted on the residual fields has a potential to yield a better reconstructed image.
We now explain the residual interpolation (RI) of [8] in more detail, with reference to FIG. 1. The input to the method is referred to as p, which is one of the red (R) or blue (B) channels of the mosaiced image. p is demosaiced under the guidance of a complete image d. In [8], the complete image d is a G-channel image obtained via the GBTF [5] method. FIG. 1 shows the RI-based reconstruction scheme for restoring the R channel (that is, p=Rm). An equivalent algorithm is performed for the B channel.
For each color channel under reconstruction, the RI process consists of two stages: (1) generating the estimated channel image and computing the corresponding residual field, and (2) interpolating the residual field, and using it for compensating the estimated channel image.
In Stage (1), the estimated image p is obtained by using a regression filter (also known as the guided filter) [9]. A regression filter is a filter which, at each given pixel of an image, modifies a first intensity value (p) at that pixel, using the first intensity values in a window centred at the given pixel, and second pixel intensity values (guide values d) of the pixels within the window. Specifically it is defined as
                                          p            _                    ⁡                      (                          i              ,              j                        )                          =                                                                              ∑                                                            (                                              u                        ,                        v                                            )                                        ∈                                          w                                              i                        ,                        j                                                                                            ⁢                                  a                  ⁡                                      (                                          u                      ,                      v                                        )                                                              MN                        ·                          d              ⁡                              (                                  i                  ,                  j                                )                                              +                                                    ∑                                                      (                                          u                      ,                      v                                        )                                    ∈                                      w                                          i                      ,                      j                                                                                  ⁢                              b                ⁡                                  (                                      u                    ,                    v                                    )                                                      MN                                              (        1        )            where wi,j is the M×N local window centered at pixel (i, j). The coefficients a(u, v) and b(u, v) are first computed at each pixel location (u, v) by performing linear regression between p and d over the M×N local window wu,v centered at (u, v). For ease of presentation, the above-described regression filtering process is denoted as: p=(p|d). Then, the residuals can be computed as Δp=p−p, involving only those known values.
In Stage (2), bilinear interpolation is applied to Δp to obtain the estimated Δp, which is used to compensate the estimated image p to generate the reconstructed image q.
As for the green colors of the demosaiced image, they are obtained using another approach, called the gradient-based threshold free (GBTF) demosaicing [5].